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ABSTRACT 



A system and apparatus for using speech recognition and 
verification to provide secure and authorized data transmis- 
sions between networked computer systems is provided. The 
system includes first and second network computer systems 
wherein a request for a transaction by user of the first 
computer system causes the user to be prompted to enter a 
spoken identifier such as a credit card number, PIN number 
or password. This spoken identifier is converted from speech 
data into speech feature data using either a resident software 
application or a downloaded application from the second 
computer system. The speech feature data is transmitted to 
the second computer system wherein speech recognition and 
verification engines identify the spoken identifier and deter- 
mine whether or not the user who spoke the identifier is 
properly associated with the spoken identifier. Upon suc- 
cessful completion of this recognition and verification 
process, the requested transaction is completed. 

24 Claims, 5 Drawing Sheets 
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SPEECH RECOGNITION AND into a microphone connected to the client unit and the 

VERIFICATION SYSTEM ENABLING spoken data comprising the identifier is converted into 

AUTHORIZED DATA TRANSMISSION OVER speech feature data. This conversion is carried out by a 

NETWORKED COMPUTER SYSTEMS locally resident software application or may be done by an 

5 application or plug-in applet that has been downloaded from 

TECHNICAL FIELD the server unit in response to the requested transaction. 

Tne present invention relates to secure data transmission, ^ s P eec i h fe ?f u f e ^ data » transmitted over a computer 

and more particularly to a method and apparatus authorizing communications link from the chent unit to the server unit 

data transmission over a computer network environment 1fl for ^ rther proce^mg Optionally, the speech feature data 

utilizing a client/server architecture. 10 ma y be securcd ^ addltlonal state of art processes, such 

as encryption, before transmission of the speech feature data 

BACKGROUND OF THE INVENTION t0 tne unit. A speech recognition engine located at the 

server unit uses the speech feature data to identify and 

The current expansion of computer network systems and confirm the spoken identifier entered by the user. The speech 

linked architectures, such as the Internet, have greatly 15 feature data is then further processed by a speech verifica- 

increased the opportunity for carrying out transactions tion engine to confirm that the user who entered the spoken 

through networked computers. Unfortunately, the perfor- identifier is in fact the user associated with the spoken 

mance of transactions via networked computers requires the identifier and is authorized to perform a requested transac- 

transmission of sensitive data (credit card numbers, PIN tion. 

numbers, passwords, etc.) over the communications network 20 AdditioaaUy> the speec h feature data for the identifier may 

interconnecting the computers. A communications network be t0 previously transmitted versions of the 

is for the most part a nonsecure transmission environment speech feature data for the identifier to determine if it 

that is subject to access by unauthorized third parties An ma tchesany of the previously transmitted versions. When an 

unauthorized party is able to get a hold of sensitive infor- exact match exists> the transmitted speech feature data ^ 

matioc i by unlawfuUy monitormg coinputer communications 25 mafked ^ ct ^ ^ { be 

over the commumcations network. This of course can be taken If the spoken identifier ^ recognized atld verified ^ 

greatly damaging to an individual or business. Also, transfer being associated with the ^ entering the identifi er and no 

of data over a communications network does not provide an questions arise due to an exact match with previously 

easy way to ensure a party is authorized to transmit or tran smitted data, the transaction request is completed, 

receive sensitive data. 30 . - ■ 

Ine foregoing has outlined some of the more pertinent 

One current method for providing secure transmission of aspects of ^ m invention . aspects should be 

sensitive data in a computer network environment relies on construe d to be merely illustrative of some of the more 

encryption of the data prior to transmission Unfortunately, promment features and applications of the invention. Many 

newly devised decryption methods and faster computer other tentfcizl results can be attained by applying the 

systems continually make it easier for unauthorized third 35 disclosed invention in a modified manner as will be 

parties to crack any encryption code, thus rendering sensi- described . Accordingly, other aspects and a fuller under- 

tive data vulnerable to unauthorized attack. Once the sen- standing of the invention may ^ had by referri to the 

™a™a i ta u decrypted by an unauthorized f 0 n owing Detailed Description of the preferred embodi- 

mdividual, this party may now use this sensitive data with- meDt 

out authorization to make purchases or carry out any number 40 

of unauthorized transactions. Since many current encryption BRIEF DESCRIPTION OF THE DRAWINGS 
methods have no mechanism for verifying the identity of the 

person submitting the sensitive information, the unautho- For a more complete understanding of the present inven- 

rized individual may continue their unlawful activities for a tion and me advantages thereof, reference should be made to 

substantial period of time. 45 me following Detailed Description taken in connection with 

Other current systems providing authentication require the acc °mpanying drawings in which: 

that additional hardware be purchased by the user and an FIG ; 1 1S a block diagram illustrating the environment of 

authentication card. The user must insert the authentication operation of the present invention; 

card into an associated card reader to access sensitive data. FIGS. 2a and 2b are flow diagrams illustrating the method 

If the card is illegally obtained by an unauthorized for providing secure communications; 

individual, this person may still access the sensitive data. FIG. 3 is a flow diagram of the speech recognition and 

Thus, a system enabling secure transmission of sensitive verification algorithms; and 

data and verification of a sender's identity would greatly FIGS ^ and 4b iUustrate manners of use for the 

benefit the expanded use of transactions occurring over of ^ e nt invention . 

networked computer systems. 

DETAILED DESCRIPTION 

BRIEF SUMMARY OF THE INVENTION 

Referring now to the drawings and more particularly to 

The present invention overcomes the foregoing and other FIG. 1, there is illustrated the general operating environment 

problems with a system and method using speech recogni- 60 of the present invention. The preferred embodiment is 

tion and verification engines to enable secure data transmis- described with respect to a client/server architecture. A client 

sions between networked computer systems. The preferred unit 10 requests a particular transaction or service from the 

embodiment of this invention utilizes a client/server archi- server unit 12 via a networked computer connection 14, such 

tecture wherein a request for a particular transaction by a as the Internet. The server unit 12 includes speech recogni- 

user at the client unit causes the server unit to prompt the 65 tion and verification engines 22 which will be more fully 

user for a spoken identifier, such as a credit card number, discussed in a moment. The client and server units may 

PIN number, password, etc. The user speaks an identifier comprise standalone computers or computer systems. 
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Tne client unit 10 includes a network browser 16, in the wherein the speech recognition and verification engines 22 

case of an Internet connection a web browser, and micro- are used to recognize the transmitted information and verify 

phone 18. The network browser 16 enables a user to navigate the user transmitting the information at step 38 has the 

between computers of the interlinked computer network authority to request a transaction using the spoken identifier, 

while the microphone 18 enables entry of spoken identifiers 5 In the alternative embodiment of FIG. 2b, a determination 

at the client unit 10. A speech feature application 20 enables is made at step 39 whether or not the spoken identifier has 

the conversion of speech data to speech feature data for been correctly recognized by the recognition algorithm. This 

transmission over the computer network to the speech may be done in a variety of ways including asking the user 

recognition and verification engines 22 of the server unit 12. to repeat the spoken identifier if a predetermined certainty 

The speech feature application 20 may be a plug-in applet, 1Q level of recognition is not achieved or by preparing speech 

application or helper application that may be downloaded feature data for the transmission back to the client unit and 

over the computer network to enable the conversion of then having this speech feature data processed by the client 

speech data to speech feature data or may be a software unit to generate a message indicating the recognized spoken 

application resident within the client unit 10. While FIG. 1 identifier. 

illustrates the speech feature application as operating within 15 Referring again to FIGS. 2a and 2b, once a positive 

the network browser 16, it should be appreciated that a speaker recognition and verification is achieved, the addi- 

helper application operating externally of the browser may tiona i procedure of comparing at step 40 the speech feature 

be used. data f or me current transmission against previous transmis- 

Referring now also to FIGS. 2a and 2b, there are illus- sions of speech feature data for the same identifier enables 

trated two flow charts describing the general method of 20 a determination of whether or not an exact match exists with 

operation of the present invention. FIG. 2a illustrates a a previous transmission. Any exact matches are marked as 

method wherein recognition of the identifier is confirmed suspect and routed to the server unit 12 for appropriate 

before transmission to the local server while FIG. 2b illus- action. This process relies on the fact that speech features 

trates a method wherein confirmation of the recognition is from one utterance to the next will be slightly different each 

performed at the remote server. The FIGS. 2a and 2b will be 25 time the utterance is spoken by the user. Comparison of the 

described together with similar steps having the same ref- utterance to previous transmissions prevents an unautho- 

erence numerals. Initially, a user at the client unit 10 makes rized user from intercepting a previous transmission and 

a transaction request at step 24 to initiate the procedure. The merely recording and replaying the information to achieve 

transactions may include a purchase, request for restricted unauthorized access. If the transmitted information and the 

data, etc. The request for a transaction initiates access at step 30 identity of the user are verified, the transaction is confirmed 

26 to the speech feature application 20. This access may at step 42 and the transaction is completed, 

entail merely opening a resident application on the client Referring now to FIG. 3, a block diagram is shown of an 

unit 10 or automatically downloading an application, helper embodiment of the voice recognition and verification algo- 

application or plug-in applet over the Internet connection 14. rimms 43 and 50 ^ functional blocks set forth in the upper 

Once the speech feature application 20 is ready, the user 35 portion of the block diagram comprise those steps which are 

is prompted to speak a key phrase or identifier at step 28. performed by the speech feature application 20 located at the 

This phrase may be a credit card number, expiration date, client unit 10. These blocks comprise speech processing 

account number, personal identification number or a pass- means for carrying out a first tier of a multistage data 

word. The speech feature application 20 transforms at step reduction process. In particular, a speech is input to the 

30 the speech data representing the phrase from the micro- 40 speech feature application 20, a feature extractor 60 extracts 

phone 18 into speech feature data used for recognition and a set of primary features that are computed in real time every 

verification by the speech recognition and verification 10 milliseconds. The primary features include heuristically- 

engines22. Conversion of the speech data to speech feature developed time domain features (e.g. zero crossing rates) 

data prior to transmission over the networked computer and frequency domain information such as fast fourier 

connection 14 is desirable due to the fact that the number of 45 transform (FFT) coefficients. The output of the feature 

bits necessary to represent speech features are smaller than extractor 60 is a reduced set (approximately 40,000 data 

the number of bits necessary to represent the speech itself. points/utterance instead of the original approximately 

However, it should be noted that the speech data may be 80,000 data points/utterance) and is applied to a trigger 

transmitted to the server unit 12 suck that transformation, routine 62 that captures spoken words using the primary 

recognition and verification all occur at the server unit. 50 features. The trigger routine 62 is connected to a secondary 

In the embodiment of FIG. 2a, the transformed speech feature routine 63 for computing "secondary features" from 

data may be initially recognized at optional step 31 to the primary features. The secondary features preferably 

confirm that the identifier can be correctly identified by the result from non-linear transformations of the primary fea- 

speech recognizer prior to transmission of the data. This step hires. The output of the routine 63 is connected to phonetic 

would use a speech recognition algorithm as will be more 55 segmentation routine 64. After an utterance is captured and 

fully described in a moment. If the data is properly recog- the secondary features are computed, the routine 64 provides 

nized at step 31, control passes off to optional step 32 for automatic phonetic segmentation. To achieve segmentation, 

further processing of the speech feature data. Otherwise, the the phonetic segmentation routine 64 preferably locates 

user is prompted to reenter the phase or identifier at step 28. voicing boundaries by determining an optimum state 

Referring back to both FIGS. 2a and 2b, in an optional 60 sequence of a two-state Markov process based on a sequence 

step, the speech feature data may be encrypted at step 32 to of scalar discriminate function values. The discriminate 

provide an additional level of security during the transmis- function values are generated by a two-class Fisher linear 

sion process. More security may be obtained by layering transformation of secondary feature vectors. The voicing 

additional levels of security on top of the encryption. The boundaries are then used as anchor points for subsequent 

order of the transmitted data may also be scrambled to 65 phonetic segmentation. 

provide an additional layer of encryption. The encrypted After the phonetic boundaries are located by the phonetic 

data is then transferred at step 34 to the server unit 12 segmentation routine, the individual phonetic units of the 
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utterance are analyzed and so called "tertiary features" are 
computed by a tertiary feature calculation routine 65. These 
tertiary features preferably comprise information (e.g., 
means or variances) derived from the secondary features 
within the phonetic boundaries. The tertiary features are 5 
used by both the voice recognition algorithm 48 and the 
voice verification algorithm 50 as will be described. The 
output of the routine 65 is a tertiary feature vector of 
approximately 300 data point/utterance. As can be seen then, 
the upper portion of FIG. 3 represents the first tier of the 
multistage data reduction process which significantly 
reduces the amount of data to be analyzed and transferred 
over the Internet connection 14 that still preserves the 
necessary class of separability, whether it is digit-relative or 
speaker-relative, necessary to achieve recognition or 
verification, respectively. The middle portion of FIG. 3 15 
represents a second tier of the data reduction process, and as 
will be described, comprises the transformation routines 49a 
and 49b occurring at the voice verification and recognition 
engines 22 of the server unit 12 (FIG. 1). 

To effect speaker independent voice recognition, the ter- 20 
tiary features are first supplied to the voice recognition linear 
transformation routine 49a. This routine multiplies the ter- 
tiary feature vector by the voice recognition feature trans- 
formation data (which is a matrix) 52a to generate a voice 
recognition parameter data factor for each digit. The output 2 5 
of the transformation routine 49a is then applied to a voice 
recognition statistical decision routine 66a for comparison 
with the voice recognition class of reference data 54. The 
output of the decision routine 66a is a yes/no decision 
identifying whether the digit is recognized and, if so, which 30 
digit was spoken. 

Specifically, a decision routine 66a evaluates a measure of 
word similarity for each of the eleven digits (zero through 9 
and "OH") in the vocabulary. The voice recognition class 
reference data 54 includes various elements (e.g., accep- 35 
tance thresholds for each digit class, inverse covariances and 
mean vectors for each class) used by the decision strategy. 
For a digit to be declared (as opposed to being rejected), 
certain acceptance criteria must be met. The acceptance 
criteria may include, but need not be limited to the follow- 40 
ing: The voice recognition algorithm determines the closest 
match between the class reference data and the voice rec- 
ognition parameter vector for the digit; this closest match is 
a so-called "first choice." The next closest match is a 
"second choice." Each choice has its own matching score. 45 
The digit is declared if (1) the matching score of the first 
choice is below a predetermined threshold, and (2) the 
difference between the matching scores of the first choice 
and the second choice digits is greater than another prede- 
termined threshold. When all words of the spoken identifier 50 
have been recognized, the voice recognition portion of the 
method is complete. 

To effect voice verification, the tertiary features are also 
supplied to a linear transformation routine 49b that multi- 
plies each tertiary feature vector (output from routine 65) by 55 
the voice verification feature transformation data (which is 
a matrix) 526. The output of the routine 49b is an N element 
vector of voice verification parameter data for each digit of 
the password, with N preferably approximately equal to 25. 
The voice verification parameter data vector is then input to 60 
a verifier routine 66b which also receives the voice verifi- 
cation class reference data for the caller. Specifically, the 
voice verification class reference data is provided from the 
voice verification reference database 55. As noted above, the 
address in the database 55 of the user's voice verification 65 
class reference data is defined by the user's password 
derived by the voice recognition algorithm 48. 
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Verifier routine 66b generates one of three different out- 
puts: ACCEPT, REJECT and TEST An ACCEPT output 
may authorize the user to from the transaction database. The 
REJECT output is provided if the verifier disputes the 
purported identify of the user. The TEST output initiates the 
prompting step wherein additional follow-up questions are 
asked to verify the user's identity. 

Referring now to FIGS. 4a and 4i>, there are illustrated 
alternative embodiments of the present invention wherein 
the speech recognition and speech verification engine soft- 
ware is distributed. In FIG. 4a, a customer 90 speaks an 
identifier which is transformed into speech feature data and 
transmitted to the merchant server 94. The merchant server 
94 runs the speech features through the recognition engine 
software 96 to recognize the identifier provided by the 
customer 90. The customer is queried once the spoken 
identifier is recognized to confirm it has been correctly 
identified. Once confirmed, the recognized identifier, the 
features for speech verification and the transaction data are 
sent to the payment gateway processor server 98 for 
verification, using speech verification engine software 100, 
of the authorization of the customer providing the spoken 
identifier to carry out a requested transaction. Once the 
identifier is verified, the payment gateway processor server 
98 transmits authorization to complete the transaction to the 
merchant server. Transmissions preferably occur over a 
secure channel such as a dedicated phone line or dedicated 
computer network between the payment gateway processor 
server 98 and the merchant server 94. Once the merchant 
server 94 obtains a successful authorization, the merchant 
completes the transaction with the customer and delivers the 
product or services. 

FIG. 4b illustrates another method wherein the merchant 
server 94 simply passes the speech feature data and trans- 
action data to the payment gateway processor server 98 so 
that recognition and verification are both accomplished by 
software engines 102 at the payment gateway processor 
server 98. This method may be used to limit the possibility 
of merchant fraud. The payment gateway processor server 
98 confirms recognition with the customer and determines 
the required response to the credit authorization requests and 
notifies the merchant of the results. Upon recognition and 
verification of authority to perform the transaction, transac- 
tion approval is transmitted to the merchant server 94. 

Use of the embodiments illustrated in FIGS. 4a and 4b 
may occur in the manner of a user speaking their credit card 
authorization number into a terminal or unit located at a 
merchant location. Alternatively, the user may enter their 
credit card number via a magnetic card reader terminal or 
key pad entry and speak an identifier such as their name that 
does not reveal any secret to an eavesdropper. 

It should be appreciated by those skilled in the art that the 
specific embodiments disclosed above may be readily uti- 
lized at a basis for modifying or designing other structures 
for carrying out the purpose of the present invention. It 
should also be realized by those skilled in the art that such 
equivalent constructions do not depart from the spirit and 
scope of the invention as set forth in the appended claims. 

What is claimed is: 

1. A system for providing secure data transmissions 
between networked computer systems, comprising: 

at least one server computer connected to a plurality of 
client computers through a computer communications 
link for receiving transaction requests and speech fea- 
ture data from the client computers; 

means within the server computer for prompting one of 
the client computers to request a spoken identifier from 
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the user of the one client computer in response to the 
transaction request; 

a recognition engine for deriving speech feature data from 
the spoken identifier; 

a verification engine for verifying the spoken identifier 5 
represented by the speech feature data is associated 
with a user speaking the identifier; and 

means for completing the transaction request upon veri- 
fication of the association of the user with the spoken 
identifier. 10 

2. The system of claim 1 wherein the speech recognition 
engine is on the server computer. 

3. The system of claim 1 wherein the verification engine 
comprises: 

a database for storing the speech feature data for previ- 15 
ously received spoken identifiers; 

a comparator for comparing the speech feature data to the 
speech feature data stored in the database for a user, 
said verification engine rejecting access when an exact 
match is detected between the speech feature data and 2Q 
the speech feature data stored in the database for a user. 

4. The system of claim 1 wherein the computer commu- 
nications link comprises the Internet. 

5. The system of claim 1 further including means for 
decrypting speech feature data received from the client 
computer. 25 

6. A method for securely transmitting data between com- 
puter systems on a computer communications link, com- 
prising the steps of: 

prompting a user at a first computer system to enter a 

spoken identifier in response to a transaction request by 30 

the user on the first computer; 
converting the spoken identifier from speech data to 

speech feature data using a speech transformation 

engine; 

transmitting the speech feature data to a second computer 
system; 

identifying the spoken identifier using a speech recogni- 
tion engine; 

verifying the spoken identifier was entered by a user 4Q 
associated with the spoken identifier using a speech 
verification engine; and 

providing the requested transaction from the second com- 
puter to the first computer if the identified spoken 
identifier was spoken by the associated user. 45 

7. The method of claim 6 further including the step of 
confirming the spoken identifier was correcdy identified by 
the speech recognition engine. 

8. The method of claim 7 further including the step of 
re-prompting the user to enter a spoken identifier if the 50 
spoken identifier is not correctly identified. 

9. The method of claim 6 further including the step of 
transferring a speech feature extraction engine to the first 
computer system in response to the transaction request. 

10. The method of claim 9 further including the step of 55 
decrypting the speech feature data prior to identifying the 
spoken identifier. 

11. The method of claim 6 further including the steps of: 
comparing the speech feature data received from the first 

computer system to previously received speech feature eo 
data to determine if any identical matches exist; and 
identifying the speech feature data as suspect if the 
received speech feature data exactly matches previ- 
ously received speech feature data. 

12. The method of claim 6 further including the step of 65 
encrypting the speech feature data prior to transmission to 
the second computer system. 
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13. The method of claim 6 further including the step of 
prompting the user at the first computer system to enter a 
first identifier containing sensitive information and wherein 
the spoken identifier comprises verifying information. 

14. A method for securely transmitting between computer 
systems on a computer communications link, comprising the 
steps of: 

prompting a user at a first computer system to speak a 
character string beginning with a first character and 
ending with a last character thereof in response to a 
transaction request by the user on the first computer; 

generating speech feature data for each spoken character 
of the character string; 

transmitting the speech feature data to a second computer 
system over the computer communications link; 

applying the speech feature data of the character string 
and voice recognition transformation data to a voice 
recognition feature transformation to generate a first set 
of parameters for each spoken character of the first 
character string, the first set of parameters for use in a 
voice recognition system; 

applying the speech feature data and voice verification 
feature transformation data to a voice verification fea- 
ture transformation to generate a second set of param- 
eters for each spoken character of the first character 
string, the second set of parameters for use in a voice 
verification system; 

recognizing the character string using the first set of 
parameters; 

verifying the user is associated with the character string 
using the second set of parameters; and 

providing the requested transaction over the computer 
communications link from the second computer to the 
first computer if the user is verified and the character 
string is recognized. 

15. The method of claim 14 further including the step of 
transferring a speech feature extraction engine to the first 
computer system in response to the transaction request. 

16. The method of claim 14 wherein the step of verifying 
further includes the steps of: 

comparing the speech feature data received from the first 
computer system to previously received speech feature 
data to determine if any identical matches exist; and 

identifying the received speech feature data as suspect if 
the speech feature data exactly matches previously 
received speech feature data. 

17. The method of claim 14 further including the step of 
encrypting the speech feature data prior to transmission to 
the second computer system. 

18. The system of claim 17 further including the step of 
decrypting the speech feature data prior to identifying the 
spoken identifier. 

19. A system for the providing secure data transmissions 
between networked computer systems comprising: 

at least one server computer including a processor for 
receiving transaction requests and speech feature data; 

a plurality of client computers connected to the at least 
one server through a computer communications link, 
each of the client computers including a processor; 

wherein the server computer prompts one of the client 
computers to request a spoken identifier from the user 
of the one client computer in response to the transaction 
request; 

a recognition engine for deriving speech feature data from 
the spoken identifier; 
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a verification engine for verifying the spoken identifier 
represented by the speech feature data is associated 
with a user speaking the identifier; and 

means for completing the transaction request upon veri- 
fication of the association of the user with the spoken s 
identifier. 

20. A server computer for use in a system for the provid- 
ing secure data transmissions between networked computer 
systems, the server computer comprising: 

a processor for receiving transaction requests and speech 10 
feature data from a plurality of client computers con- 
nected to the server through a computer communica- 
tions link, 

wherein the server computer prompts one of the client J5 
computers to request a spoken identifier from the user 
of the one client computer in response to the transaction 
request; 

a verification engine for verifying the spoken identifier 
associated with a user speaking the identifier. 2 o 

21. The server computer of claim 20 further including a 
recognition engine for deriving speech feature data from the 
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spoken identifier and wherein the verification engine used 
the speech feature data for verifying the spoken identifier. 

22. The server computer of claim 21 further including 
means for completing the transaction request upon verifica- 
tion of the association of the user with the spoken identifier. 

23. A client computer for use in a system for the providing 
secure data transmissions between networked computer sys- 
tems comprising: 

a processor connected to at least one server computer 
through a computer communications link, the processor 
for sending a transaction requests and speech feature 
data to the at least one server computer and for receiv- 
ing a request from the server computer for a spoken 
identifier from a user of the client computer in response 
to the transaction request. 

24. The client computer of claim 23 further including a 
recognition engine for deriving speech feature data from the 
spoken identifier. 

* * * * * 
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